Dynamic Hand Gesture Recognition Research Articles

Decoupled and boosted learning for skeleton-based dynamic hand gesture recognition

With the development of cost-effective depth sensors, skeleton-based dynamic hand gesture recognition has made significant progress. Existing methods mostly utilize a single model to learn all spatial–temporal features. Meanwhile, they cannot effectively boost key features and make use of multi-scale features. In this paper, we propose a lightweight dual-stream framework, which consists of a temporal mutual boosted stream (TMB-Stream) and a spatial self-boosted stream (SSB-Stream). In the TMB-Stream, we design a hybrid attention module (HAM) to boost important motion features from temporal sequences, which is composed of a multi-scale multi-head attention module (MMAM) and a spatial–temporal attention module (STAM). In the SSB-Stream, we present a self-boosted learning manner to promote the performance of the spatial stream. Specifically, we design a multi-scale auto-encoder (MAE), which can use limited skeleton data to extract and boost spatial latent features by minimizing the gap between original and reconstructed skeleton images. In addition, we propose a multi-scale fusion module (MFM) to effectively fuse multi-scale features in stages. Experimental results show that our lightweight framework yields satisfactory performance on SHREC’17 Track and DHG-14/28 datasets, as well as very competitive performance on FPHA dataset.

Motion feature estimation using bi-directional GRU for skeleton-based dynamic hand gesture recognition

Motion feature estimation using bi-directional GRU for skeleton-based dynamic hand gesture recognition

A Review on Sign Language Recognition and Learning System

This paper presents innovative approaches to various aspects of sign language recognition and learning, catering to the diverse needs within this field. It introduces a system utilizing 2D image sampling and concatenation, trained with convolutional neural networks, achieving accurate and robust sign recognition even with low-cost cameras. Another system, SignQuiz, offers a cost-effective web-based solution for learning finger-spelled signs in Indian Sign Language (ISL), outperforming traditional printed mediums. Additionally, a dynamic hand gesture recognition system employing deep learning architectures demonstrates superior performance over existing methods, addressing challenges such as hand segmentation and sequence modeling. Furthermore, an efficient deep convolutional neural network approach for hand gesture recognition, including transfer learning, achieves high recognition rates across various datasets. Finally, a robust model for static sign recognition in ISL, utilizing CNNs and evaluating on a diverse dataset, attains exceptional accuracy and outperforms previous works. These contributions collectively advance the field of sign language recognition and learning, offering solutions that are accurate, cost-effective, and efficient, thereby facilitating better communication and interaction for the hearing-impaired community. Key Words: CNN, Sign language recognition, Hand gesture recognition, Machine learning, Transfer Learning, SVM.

DC-NAS: Divide-and-Conquer Neural Architecture Search for Multi-Modal Classification

Neural architecture search-based multi-modal classification (NAS-MMC) methods can individually obtain the optimal classifier for different multi-modal data sets in an automatic manner. However, most existing NAS-MMC methods are dramatically time consuming due to the requirement for training and evaluating enormous models. In this paper, we propose an efficient evolutionary-based NAS-MMC method called divide-and-conquer neural architecture search (DC-NAS). Specifically, the evolved population is first divided into k+1 sub-populations, and then k sub-populations of them evolve on k small-scale data sets respectively that are obtained by splitting the entire data set using the k-fold stratified sampling technique; the remaining one evolves on the entire data set. To solve the sub-optimal fusion model problem caused by the training strategy of partial data, two kinds of sub-populations that are trained using partial data and entire data exchange the learned knowledge via two special knowledge bases. With the two techniques mentioned above, DC-NAS achieves the training time reduction and classification performance improvement. Experimental results show that DC-NAS achieves the state-of-the-art results in term of classification performance, training efficiency and the number of model parameters than the compared NAS-MMC methods on three popular multi-modal tasks including multi-label movie genre classification, action recognition with RGB and body joints and dynamic hand gesture recognition.

Dyhand: dynamic hand gesture recognition using BiLSTM and soft attention methods

Dyhand: dynamic hand gesture recognition using BiLSTM and soft attention methods

Through deep learning, dynamic hand gesture recognition of sign language learning algorithm

Through deep learning, dynamic hand gesture recognition of sign language learning algorithm

Multimodal fusion hierarchical self-attention network for dynamic hand gesture recognition

Recent improvements in dynamic hand gesture recognition have seen a shift from traditional convolutional architectures to attention-based networks. These attention networks have been proven to outclass CNN + LSTM architectures, showing higher accuracy as well as reduced model parameters. Especially, skeleton-based attention networks have been shown to outperform visual-based networks due to the rich information from skeleton-based hand features. However, there is an opportunity to introduce complementary features from other modalities like RGB, depth, and optical flow images to enhance the recognition capability of skeleton-based networks. This paper aims to explore the addition of a multimodal fusion network to a skeleton-based Hierarchical Self-Attention Network (MF-HAN) and test for increased model effectiveness. Unlike traditional fusion techniques, this fusion network uses features derived from other sources of multimodal data in a reduced feature space using a cross-attention layer. The model outperforms its root model and other state-of-the-art models on the SHREC’17 track dataset, especially in the 28 gestures setting by more than 1 % in gesture classification accuracy. The experimentation was tested on the DHG dataset as well.

Survey on vision-based dynamic hand gesture recognition

Survey on vision-based dynamic hand gesture recognition

An Efficient Graph Convolution Network for Skeleton-Based Dynamic Hand Gesture Recognition

Dynamic hand gesture recognition has evolved as a prominent topic of computer vision research due to its vast applications in human-computer interaction, robotics, and other domains. Although there are numerous related recognition studies, the State-Of-The-Art (SOTA) methods are overparametrized. Specifically, the number of model parameters is quite large, which results in high computational costs. This work, referring to Songs ResGCN, designs an efficient and lightweight graph convolutional network, named ResGCNeXt. ResGCNeXt learns rich features from skeleton information and achieves high accuracy with less number of model parameters. First, three data preprocessing strategies according to motion analysis are designed to provide sufficient features for the recognition model. Then, an efficient graph convolutional network structure combining bottleneck and group convolution is designed to reduce the number of model parameters without loss of accuracy. Furthermore, an attention block called SENet-Part attention (SEPA) is added to improve channel and spatial feature learning. This study is validated on two benchmark datasets, and the experimental results show that ResGCNeXt provides competitive performance, especially in significantly reducing the number of model parameters. Compared to HAN-2S, which is one of the best SOTA methods, our method has half model parameters and a 0.3% higher recognition rate.

Dynamic Hand Gesture Recognition for Indian Sign Language using Integrated CNN-LSTM Architecture

Human Centered Computing is an emerging research field that aims to understand human behavior. Dynamic hand gesture recognition is one of the most recent, challenging and appealing application in this field. We have proposed one vision based system to recognize dynamic hand gestures for Indian Sign Language (ISL) in this paper. The system is built by using a unified architecture formed by combining Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM). In order to hit the shortage of a huge labeled hand gesture dataset, we have created two different CNN by retraining a well known image classification networks GoogLeNet and VGG16 using transfer learning. Frames of gesture videos are transformed into features vectors using these CNNs. As these videos are prearranged series of image frames, LSTM model have been used to join with the fully-connected layer of CNN. We have evaluated the system on three different datasets consisting of color videos with 11, 64 and 8 classes. During experiments it is found that the proposed CNN-LSTM architecture using GoogLeNet is fast and efficient having capability to achieve very high recognition rates of 93.18%, 97.50%, and 96.65% on the three datasets respectively.

A Survey of Sign Language Recognition

Sign Language is mainly used by deaf (hard hearing) and dumb people to exchange information between their own community and with other people. It is a language where people use their hand gestures to communicate as they can't speak or hear. The goal of sign language recognition (SLR) is to identify acquired hand motions and to continue until related hand gestures are translated into text and speech. Here, static and dynamic hand gestures for sign language can be distinguished. The human community values both types of recognition, even if static hand gesture recognition is easier than dynamic hand gesture recognition. By creating Deep Neural Network designs (Convolution Neural Network designs), where the model will learn to detect the hand motions images throughout an epoch, we are using Deep Learning Computer Vision to recognize the hand gestures. After the model successfully recognizes the motion, an English text file is created that can subsequently be translated to speech. The user can choose from a variety of translations for this paragraph. This application can be used without an internet connection and is entirely offline. With this model's improved efficiency, communication will be easier for the deaf (hard of hearing) and disabled people. We shall discuss the use of deep learning for sign language recognition in this paper. Key Words: sign language, convolutional neural network, computer vision.

Realtime Recognition of Dynamic Hand Gestures in Practical Applications

Dynamic hand gesture acting as a semaphoric gesture is a practical and intuitive mid-air gesture interface. Nowadays benefiting from the development of deep convolutional networks, the gesture recognition has already achieved a high accuracy, however, when performing a dynamic hand gesture such as gestures of direction commands, some unintentional actions are easily misrecognized due to the similarity of the hand poses. This hinders the application of dynamic hand gestures and cannot be solved by just improving the accuracy of the applied algorithm on public datasets, thus it is necessary to study such problems from the perspective of human-computer interaction. In this article, two methods are proposed to avoid misrecognition by introducing activation delay and using asymmetric gesture design. First the temporal process of a dynamic hand gesture is decomposed and redefined, then a realtime dynamic hand gesture recognition system is built through a two-dimensional convolutional neural network. In order to investigate the influence of activation delay and asymmetric gesture design on system performance, a user study is conducted and experimental results show that the two proposed methods can effectively avoid misrecognition. The two methods proposed in this article can provide valuable guidance for researchers when designing realtime recognition system in practical applications.

A novel static and dynamic hand gesture recognition using self organizing map with deep convolutional neural network

Gesture recognition has gained a lot of popularity as it allows humans to communicate with real or virtual systems through gestures, offering new and natural interaction modalities. Recent technologies, such as augmented reality (AR) and the Internet of Things (IoT), have witnessed enormous growth in computer applications that focus on human–computer interaction (HCI). However, a few of these tactics make use of a combination of methods, such as image segmentation, pre-processing, and classification. The hessian-based multiscale filtering and YCbCr colour space are used to separate the gesture region to be recognized. A modified marker-controlled watershed method is employed to segment the gesture contour along with the eight-connector graph to increase recognition precision. The proposed hand gesture recognition methodology uses Self Organizing Map (SOM) with Deep Convolutional Neural Network (DCNN) provides better results with fast convergence speed. Experiments were carried out on a dataset of 30 static and 6 dynamic gestures and also evaluated on a publicly available IIITA-ROBITA ISL Gesture Database to show the effectiveness. The results show that the suggested method can recognize gesture classes with 95.63% accuracy rate without significantly affecting the recognition time. The proposed algorithm was then implemented to control household appliances.

Novel error correction-based key frame extraction technique for dynamic hand gesture recognition

Novel error correction-based key frame extraction technique for dynamic hand gesture recognition

Snapture—a Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition

As robots are expected to get more involved in people’s everyday lives, frameworks that enable intuitive user interfaces are in demand. Hand gesture recognition systems provide a natural way of communication and, thus, are an integral part of seamless human-robot interaction (HRI). Recent years have witnessed an immense evolution of computational models powered by deep learning. However, state-of-the-art models fall short of expanding across different gesture domains, such as emblems and co-speech. In this paper, we propose a novel hybrid hand gesture recognition system. Our Snapture architecture enables learning both static and dynamic gestures: by capturing a so-called snapshot of the gesture performance at its peak, we integrate the hand pose and the dynamic movement. Moreover, we present a method for analyzing the motion profile of a gesture to uncover its dynamic characteristics, which allows regulating a static channel based on the amount of motion. Our evaluation demonstrates the superiority of our approach on two gesture benchmarks compared to a state-of-the-art CNNLSTM baseline. Our analysis on a gesture class basis unveils the potential of our Snapture architecture for performance improvements using RGB data. Thanks to its modular implementation, our framework allows the integration of other multimodal data, like facial expressions and head tracking, which are essential cues in HRI scenarios, into one architecture. Thus, our work contributes both to integrative gesture recognition research and machine learning applications for non-verbal communication with robots.

Real-time continuous detection and recognition of dynamic hand gestures in untrimmed sequences based on end-to-end architecture with 3D DenseNet and LSTM

Real-time continuous detection and recognition of dynamic hand gestures in untrimmed sequences based on end-to-end architecture with 3D DenseNet and LSTM

A novel sEMG-based dynamic hand gesture recognition approach via residual attention network

A novel sEMG-based dynamic hand gesture recognition approach via residual attention network

SBI-DHGR: Skeleton-based intelligent dynamic hand gestures recognition

Hand gesture recognition (HGR) plays a significant role in interpreting the meaning of sign language, human–computer interaction, and robot control. This paper proposes a real-time skeleton-based intelligent dynamic hand gesture recognition (SBI-DHGR) approach, which comprises three modules: Palm Centroid (PC), Data augmentation with the Tenet Effect, and Deep learning architecture. The palm Centroid (PC) module is introduced to identify the proposed palm center joint. Data augmentation with a tenet effect module is designed to improve the CNN model’s generalizability. Further, a novel deep learning model is proposed for temporal 3D-HGR by exploiting the capabilities of a multi-channel convolutional neural network (CNN) and long short-term memory (LSTM) recurrent network. The multi-channel CNN is introduced to learn the cardinal positions of hand joints by capturing the low, average, and high-level features. LSTM is embedded to learn the temporal characteristics of hand joints. The effectiveness of the SBI-DHGR framework is evaluated over five challenging datasets: SHREC-14, SHREC-28, DHG-14, DHG-28, and FPHA, by adopting person-dependent and person-independent validation setups.

Textural feature descriptors for a static and dynamic hand gesture recognition system

Textural feature descriptors for a static and dynamic hand gesture recognition system

Contactless System Navigation Using Dynamic Hand Gesture Recognition

Abstract: Hand gestures are a natural way for human-robot interaction. Vision based dynamic hand gesture recognition has become a hot research topic due to its various applications. Traditional (or conventional) methods of human-computer interaction, namely, keyboards, pointing devices, and similar physical input tools, and the more popular and commonly used touchscreens, are only as useful through the application of pressure through physical touch. The recent pandemic has driven a major argument about the substantial and often, indispensable need for contact-less operation of systems, irrespective of the scale of the industry. Say for example, supply chain systems that are handled by scores of employees in the span of a day. It is quite fundamental knowledge now, that how public systems that work on physical contact can pose as potential vectors for diseases. Hand gesture recognition can turn out to be an excellent solution to this problem; facilitating truly contact-less interaction with computing devices, and at the same time providing dependable and respectable efficiency. This approach will be further elaborated in upcoming editions of this research article.